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Description 

The present inventbn relates to packet switched communication systems and. rrwre particularly, to traffic shaping 
for causing the time multiplexed packet flows at queuing points within such systems or system elements to conform 
5 to specified traffic descriptors. 

A. Traffic Contracts/Definitions 

Most applications that are currently running on packet switched communication networks can functk>n acceptably 

10 with whatever bandwidth they happen to obtain from the network because they have 'elastic" bandwidth requirements 
The service classes that support these applications is known as 'best efforts" service in the Internet community and 
as 'Available Bit Rate' (ABR) in the Broadband ISDN/ATM community. 

There is. however, a growing demand for network services that provide bounded jitter or, in other words, bounded 
packet delay variation {commonly referred to as cell delay variation in an ATf^ context). For example, this type of 

?5 service is required for real time applications, such as circuit emulation and video. It is not clear whether and how the 
mtemet community will respond to this demand, but the Broadband ISDN/ATM community has responded by introduc- 
ing the notion of a user-network negotiated traffic contract. 

As is known, a user-network AThA contract is defined by a traffic descriptor which includes traffic parameters, 
tolerances and quality of sen^ice requirements. A confornr^ce definition is specified for each of the relevant traffic 

20 parameters. Accordingly. ATM services may make use of these traffk; parameters and their corresponding conformance 
specifications to support different combinations of Quality of Sen/ice (QoS) objectives and multiplexing schemes. 
Partially overlapping sets of ATM traffic classes have been defined by the Telecommunications Standardizatk>n Sector 
of the International Telecommunications Union (ITU-T) and the ATM Forum. In some instances, traffic classes which 
have essentially identical attributes have been given different names by these two groups, so the foltowing name 

25 translation table identifies the existing equivalent counterparts: 



ITU-T Traffic Class 


ATM Forum Traffic Class 


ABR 

Deterministic Bit Rate (DBR) 
Statistical Bit Rate (SBR) 
(No existing counterpart, but understudy) 
(No existing counterpart, but understudy) 


ABR 

Constant Bit Rate (CBR) 
Variable Bit Rate (VBR) 
Real time Variable Bit Rate (rt-VBR) 
Unspecified Bit Rate (UBR) 



An ATM service contract for a virtual circuit (VC) connection or a virtual path (VP) connection may include multiple 
parameters describing the sen/ice rate of the connection. This includes the Peak Cell Rate (PGR), the Sustainable 
Cell Rate (SCR) the Intrinsic Burst Tolerance (IBT). and the Minimum Cell Rate (MCR). Not all of these parameters 
are relevant for every connection or every servwe class, but when they are implied or explcitty specified elements of 
the service contract, they must be respected. VC connections are the primary focus of the following discussion, but it 
will be understood the VP connections can also be so specified. The data transport unit for an ATM connectbn usually 
is referred to as a 'cell.' In this disclosure, however, the temn 'packet' sometime is used to refer to the data transport 
unit because this more general tenmtnology is consistent with some of the broader aspects of the innovations. 

The Generic Ceil Rate Algorithm (GCRA), which is specified in ITU-T Recommendatkxi 1.371, is well suited for 
testing a packet or cell flow for conformance with a traffic descriptor. To perform such testing, the GCRA requires the 
specification of an emission interval (i.e., the reciprocal of a flow rate) and a tolerance, x. In practice, this tolerance 
may depend on a variety of factors, including the connection, the connection setup parameters, or the class of sen/ice. 
As will be seen, the GCRA can be employed as a Boolean functbn, where for a flow of fixed size packets or cells on 
a connection, the GCRA (emission inten/al, tolerance) is false if the flow is conforming to a peak rate or true if the flow 
is conforming to a minimum rate. For example, a source of cells conforms to a PGR if GCRA (1/PCR, tp^R ) is false. 
Likewise, a connection or flowconforms toan MCR if GCRA (1/MCR, t,^cr) 'S ^a'se. As will be appreciated the "emission 
inien/al" is the reciprocal of the 'cell rate.' 

A DBR traffic contract is appropriate for a source which establishes a connection in the expectation that a static 
amount of bandwidth will be continuously available to the connection throughout its lifetime. Thus, the bandwidth the 
network commits to a DBR connection is characterized by a PCR value. Further, the cell or packet flow on such a 
connection complies with the traffic contract if it conforms to GCRA (1/PCR, tpcr). On the other hand, an SBR traffic 
contract is suitable for an application which has known traffic characteristics that allow for an informed selectbn of an 
SCR and x^Qy, as well as a PCR and Tp^R. An SBR or rt-SBR flow complies with its traffic contract if the flow not only 
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conforms to GCRA (1/PCR, Tpcp). but also to GCRA (1/SCR, t,bj). 

As previously indicated, an ABR traffic contract is appropriate for applications that can tolerate the dynamic vari- 
ations in the informatiori transfer rate that result from the use of unresen/ed bandwidth. A PGR and an MCR are specified 
by the source establishing such a connection, and these parameters may be subject to negotiation with the network. 
Thus, the bandwidth that is available on an ABR connection is the sum of the MCR (which can be 0) and a variable 
ceil rate that results form a sharing of unreserved bandwidth among ABR connections via a defined allocation policy 
(i.e.. the bandwidth a source receives above its specified MCR depends not only on the negotiated PGR. but also on 
network policy). Feedback from the network enables the source application to dynamically adjust the rate it feeds cells 
or packets into an ABR connection. An ABR flow always complies with its traffic contract if it conforms to GCRA (1/MCR, 
^mcr)' 's always non-compliant if ft does not conform to GCRA (1/PCR, tp^R). Confomnance in the region between 
MCR and PGR is dependent on the ABR feedback and is thus dynamically determined. 

A UBR traffic contract is similar to the ABR contract, except that the UBR contract does not accomrrwxjate the 
specification of an MCR and has no dynamic conformance definition. Therefore, a UBR flow complies with its traffic 
contract if it confomns to GCRA (1/PCR, tp^p). 

B. Traffic Shaping 



ITU-T Recommendation 1 .371 addresses the possibility of reshaping trafTc at a network element for the purpose 
of bringing the traffic into conformance with a traffic descriptor in the following terms: 
20 "Traffic shaping is a mechanism that alters the traffic characteristics of a stream of cells on a VCC or a VPC to 

achieve a desired modificatkxi of those traffic characteristics, in order to achieve better network efficiency whilst meeting 
the QoS objectives or to ensure conformance at a subsequent interface. Traffic shaping must maintain celt sequence 
integrity on an ATM connection. Shaping modifies traffic characteristics of a ceil flow with the consequence of increasing 
the mean cell transfer delay. 

2S Examples of traffic shaping are peak cell rate reduction, burst length limiting, reduction of CDV by suitably spacing 

cells in time and queue service schemes. 

It is a network operator's choce to determine whether and where traffic shaping is performed. As an example, a 
network operator may choose to perform traffic shaping in conjunction with suitable UPC/NPC functions 
It is an operator's option to perform traffic shaping on separate or aggregate cell fiov^. 
30 As a consequence, any ATM connectwn may be subject to traffic shaping. 

The options available to the network operator/senrice provider are the following : 

a. No shaping 

3S - Dimension the network in order to accommodate any flow of conforming cells at the network ingress whilst 

ensuring conformance at the network egress without any shaping functkxi. 

b. Shaping 

^0 - Dimension and operate the network so that any flow of conforming cells at the ingress is conveyed by the 

network or network segment whilst meeting QoS objectives and apply output shaping the traffic in order to 
meet conformance tests at the egress. 

Shape the traffic at the ingress of the network or network segment and allocate resources according to the 
traffic characteristics achieved by shaping, whilst meeting QoS objectives and subsequent conformance tests 
at the network or network segment egress. 

Traffic shaping may also be used within the customer equipment or at the source in order to ensure that the cells 
generated by the source or at the UNI are conforming to the negotiated traffic contract relevant to the ATC that is used 
(see Section 5.5).* ITU-T Recommendatk>n 1.371. Sectbn 6.2.5. 

C. Scheduling for Real Time and Non- Real Time Connections/Exieting Toole and Techniquee 



As is known, if bandwidth is not divided fairly' between applications employ tng 'best efforts" Intemet service or 
ABR ATM service a variety of undesirable phenomena may occur See Lefelhocz, Lyies, Shenker and Zhang, "Con- 
ss gestion Control for Best-Effort Service: Why we need a new paradigm," IEEE Network . January/February 1996. for 
further details on mechanisms for best effort/ABR traffic. 

Most ATM switches currently are implemented with FIFO queuing. FIFO queuing exhibits pathological behaviors 
when used for ABR traffic (see "On Traffic Phase Effects in Packet -Switched Gateways' . Sally Ftayd and Van Jacobson, 
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Intemetwofkino: Researc h and Experience. Vol. 3. pp. 115-156 (1992). and "Observations on the Dynamics of a Con- 
gestion Control Algorithnn: The affects of Two-Way Traffic", Lixia Zhang. Scott Shenker, and David Clark. ACMSiocomm 
91 Conference. September 3-6. 1991. Zurich. Switzerland, pp. 133-148.). FIFO also is unable to protect correctly 
behaving users against misbehaving users (it does not provide isolation). As a result of these deficiencies non-FIFO 

5 queuing mechanisms such as weighted fair queuing (see, for example. A. Demers. S. Keshave. and S. Shenker, "Anal- 
ysis and Simulation of a Fair Queuing Algorithm." Proceedings of ACM Siqcomm. oaoes 1-12. September 1989: and 
A.K. Parekh "A Generalized Processor Sharing Approach to Flow Control in Integrated Sen/ice Networks," Ph D Thesis, 
Department ol Electrical Engineering and Computer Science. MIT 1992.) or approximations to fair queuing such as 
round-robin (Ellen L. Hahne. 'Round-robin Scheduling for Max-Min Faimess in Data Networks." IEEE Journal on Se- 
lected Areas in Communications. Vol. 9, pp. 1024-1039, Sept. 1991.) are often suggested. 

Service classes which have inelastic bandwidth requirements often require that data be transmitted through the 
network with bounded jitter (i.e.. bounded cell or packet delay variation). As shown by the above referenced Parekh 
paper, weighted fair queuing can be used to provide bounded jitter for real time streams. Moreover, Parekh's results 
have recently {Pawan Goyal. Simon S. Lam and Harrick M. Vin. "Determining End-to-End Delay Bounds in Heteroge- 

'5 neous Networks,' Proceedinos of The 5th International Workshop on Network and Operatino Svstem Support for Dtoital 
Audio and Video (NOSSDAVV Durham, N.H., April 18-22. 1995.) been extended to prove delay bounds for systems 
using the closely related mechanisms of Virtual Ckxk (Lixia Zhang. 'Virtual Clock: A New Traffic Ccwitrol Algorithm for 
Packet Switching Networks," Proceedings of ACM Siqcomm. pages 1 9-29. August 1 990.) and Self-clocked Fair Queu- 
ing (S.J. Golestani. "A Self-Clocked Fair Queuing Scheme for High Speed Applications," Proceedings of INFOCOM. 

20 pp. 636-646, 1994). 

Thus, it is known that both elastic (Best effort/ ABR) and inelastic (or real-time) servkies can benefit from the use 
of fair queuing and related algorithms. 

1. Weighted Fair Queuing and Virtual Clocl( 

2S 

Fairqueuing and related algorithms (e.g., frame-based fair queuing, deficit round robin, etc.) operateon sequences 
of packets or other data transport units (e.g.. an ATM cell is a packet for the purposes of this discusskjn). For ATM 
these sequences are identified by either the VCI or the VPI, while in the Internet protocol suite the identificatbn is on 
the basis of <IP address, protocol, port> triples (IPv4) or flow identifiers (IPv6), In both self-clocked weighted fair 
30 queuing and virtual clock, packets are ordered (sorted) by timestamps (schemes such as round-robin provide approx- 
imations to ordering of packets by timestamps). These timestamps represent the virtual finishing time (or equivalently 
the virtual starting time for the packet and are computed by taking a starting time value and adding an offset obtained 
by multiplying the length of the packet by a weight whk:h represents the particular packet sequence's share of the 
bandwidth. 

35 More particularly for virtual clock the virtual finishing time is computed as: 

VT(f. 0) = 0 

40 = rnax{Arrival(f. j+1), VT(f, j)} + Length(f, j+1)/Rate(f) (1) 

where: VT(f, j) is the virtual finishing time associated with packet j of flow (virtual circuit) f; 
Arrrval(f , j) is the arrival time of packet j of flow f ; and 
Length(f, j) is the length of packet j of flow f. 

45 

Self -clocked weighted fair queuing assigns virtual finishing times according to the formula: 

VT(f. 0) = 0 

so 

VT(f , j+1 ) = max{SystemVirtualTime. VT(f . j)} + Length(f j+1 ) * weight(f) (2) 

where: System VirtuatTime is the virtual time associated with the packet being served (being output) at the time packet 
(f, j+1) arrives. 

55 For ATM the packet length is constant because the cells are of flxed size (i.e., 53 bytes tong). Consequently, 

rightmost term in both Expression (1 ) and Expressran (2) becomes a per flow constant. For virtual clock the simplified 
expression is: 
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VT(f. j+1 ) = max{Arrival(f. j+1 ). VT(f . j)} + constant(f) (3) 
For self-clocked weighted fair queuing, on the other hand, the simplified expression is: 

5 

VT{f. j+1 ) = max{Syst8mVirtuamme. VT(f. j)} + constant (0 (4) 

in other words, an ATM queuing point which implements either virtual clock or self-clocked weighted fair queuing 
10 performs the following steps: 

1) compute the maximum of (a) the current virtual time for the VC. and (b) either of i) the amval time of the cell or 
ii) the system virtual time. 

2) add to the results of step 1 above a per-VC constant representing that VC*s share of the bandwidth, 

'5 3) service cells (transmit them) in order of increasing values of the virtual time stamps assigned by steps 1 and 2. 

2. Priority 

Giving priority to one traffic class over another means that if the higher prcrrty traffic class has cells ready for 
20 transmisskjn, those cells are always transmitted in preference to the celts of the lower priority traffic class. 

Priority mechanisms can be either preemptive or non-preemptive. This terminology comes from the operating 
system literature. A non-preemptive priority mechanism assigns a priority to an object (a process in the operating 
system world, a VC in the ATM worid) at a scheduling time, and the object then retains this priority until it is served. 
Preemptive priority mechanisnr>s, on the other hand, can change the priority of objects while they are waiting to be 
2S served For example, in a preemptive system one could say "schedule this VC with priority 3 but if it is not served within 
200 microseconds then increase its priority up to 2.* ^ 

3. Work Conserving and Non-work Conserving Queuing -rr 

30 Kleinrock. Queuing Systems. Vol. 2: Computer Aoplications. John Wiley & Sons. N.Y.. N Y. 1996. p. 113 uses the 

terminology 'work conserving' to denote any queuing system in whrch work is neither created nor destroyed. In keeping 
with this terminology a switch which, when given queued cells, always transmits cells on the outgoing link is a Nvork 
conserving switchV Switches employing a pure FIFO, weighted fair queuing or virtual clock scheduling algorithm are 
all work conserving. In contrast, a non-work conserving switch nray choose not to send cells, even when cells are 

35 queued for transmission. As will be seen, a method of doing this is to program the switch to wait until the current time 
is equal to or greater than the timestamp associated with a particular cell before transmitting that cell. 

Work conserving switches attempt to fully utilize the transmissbn link, but do not necessarily remove or prevent 
bursts. In contrast, non-woric consenting swrtches can strategically delay cells so as to re-shape traffic to rrjeet a more 
stringent conformance test (i.e.. a GCRA with a smaller t). Additionally, a non-wortt consen/ing switch in which a given 

40 connection is only allocated a specified amount of buffering can perform a policing function (in ITU temns a UPC/NPC) 
by discarding or tagging cells which overflow the allotted buffer space. An example of a non-work conserving queuing 
system is the Stalled Virtual Ckxk (Sugih Jamin, "Stalled Virtual Clock" worthing note, Department of Computer Science. 
UCLA, March 21 , 1 994). which is an adaptatk)n of Lixia Zhang's Virtual Ckx;k algorithm where virtual time is not allowed 
to run faster (it stalls or goes non-work conserving) than real-time. Also see, work by Scott Shenker that ts available 

45 by FTP at FTPPARC.XEROX.com. 

4. Calendar Queues 

A calendar queue is a time ordered list of actions, each of which is dequeued and executed when real-time is equal 
so to or greater than the time associated with the action. Calendar queues with bounded time intervals can be represented 
as a linear array which is known as a time-wheel" or "time-line." Time-wheels assign events to buckets relative to a 
pointer, where the bucket index is calculated using arithmetk; modulo the wheel size. These data structures are well 
known in the literature as a queuing mechanism. In a time-wheel, absolute time is represented as an offset relative to 
the cun-ent time ("real time"), and each element in the array is a bucket whk;h contains one or more actk>ns (typically 
55 in linked-list form) which are to be executed at the time assigned to the bucket in which they reside. Any of the buckets 
of such a time-wheel can be empty, i.e.. have no events associated with it. 

For every time-wheel, there are two times of interest: t^i^at ^te»f which correspond to the head and tail 
pointers for the active entries in the array; where i^^j^ is the time of the next entry (e.g., packet or cell) to be serviced. 



5 



BNSOOCID: <EP 0817433A2_I_> 



EP 0 817 433 A2 

tiat«t 'sthe time associated with the latest (most distant in time) bucket containing a scheduled event The difference 
between \^^^, and t,^,„, cannot be greater than the length of -the time-wheel, b. minus 1. This can be ensured by 
viewing the time as be.ng kept modulo b. and by then ensuring that no offset (the packet length multiplied by either 
the rate or the weight in virtual clock or weighted fair queuing respectively) is greater than b-1 . For an ATM link running 
5 at OC-3 speeds (149.76 mbps - the SONET payload rate) there are approximately 353208 cells/sec on the link Ac 
cordingV. rf 64Kbps (voice telephony rates) flows (approximately 1 74 cells/sec when AAL type 1 is used) are the lowest 
speed connections that need to be supported, then the ratio of the highest supported rate to the lowest rate is 2029 
which rounds up to This ratio is the maximum offset that will get added during the calculation of virtual times' 
Therefore, a ttme-wheel of length 2030 (2048 to altow for rounding up to a power of two) is sufficient to encode the 
10 virtual times associated with circuits ranging in rates from 64Kbps to full OC-3 link rate 

The length of a time-wheel array can be decreased by permitting an array element to contain more than one time 
offset. For example, if the above^escribed time-wheel is reduced to 256 elements from 2048. then each bucket wouW 
have eight time offsets mapped into it. Actions within a single bucket that spans multiple offsets may be performed out 
of order, but between buckets actions will stay in order This reduces the amount of memory that needs to be allocated 
to such a time-wheel at the cost of reducing the precisk)n of the ordering of actksns in the calendar queue. 

D. Traffic Shaping for Time Multiplexed Flows on Multiple Output Channels 

Preferably, any traffic shaping that Is needed to bring time multiplexed packet or cell flows into conformance with 
their traffic contracts is performed after the completion of all switching or routing operations that are required to separate 
flows for different output channels from each other This permits the throughput efficiency of the multiplexer to be 
optimized. 

However, prior output queued ATM switches generally have employed FIFO (First In - First Out) output buffers 
These buffers are not capable of participating in a controlled reshaping of any of the flows that pass through them 
Instead, the per-VC time multplexed flows that are output by these buffers essentially are time multiplexed composites 
of the input flows that are k>aded into them. Of course, these output flows are time delayed relative to the input flows 
because of the inherent latency of the buffers. Moreover, the cell delay variation (CDV) of one or more of these output 
flows may be increased if scheduling conflicts occur among the data transport limits of the different flows because 
these conflicts cause so-called 'transmit collisions/ 

As will be appreciated, increased CDV is especially troublesome for traffic, such as DBR traffic, which generally 
has a relatively tight tolerance. Thus, if each hop between a source and a destination includes a simple FIFO output 
queue of the foregoing type, it may be necessary to limit the number of hops this CDV sensitive traffic is permitted to 
make in order to ensure compliance within its specified tolerance. 

Accordingly, will be evident that there is a need for more efficient and more effective traffic shaping mechanisms 
and processes for ATM switches and other routers that rout traffic from multiple inputs to multiple outputs for time 
multiplexed output emission. 

The present invention meets this need by providing rate shaping in per-flow output queued routing mechanisms 
for available bit rate (ABR) servrce in networks having segmented ABR control loops. The present invention is defined 
in the appended claims. 

The present invention will be described further by way of examples, with reference to the accompanying drawings 
in which: 
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30 
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Fig. 1 is a simplified bkxk diagram of an ATM switch in which the present invention may be used to advantage; 
Fig. 2 diagramnnatrically tracks the various shapes an ATM cell may suitably take while traversing the switch shown 
^5 in Fig. 1; 

Fig. 3 is a more detailed block diagram of a representative channel on the output or transmit side of the chip shown 
in Fig. 1; 

Fig. 4 schematically illustrates a stalled virtual clock calendar queue implementation of the present invention; 
Fig. 5 schematically illustrates another stalled virtual clock calendar queue implementation of the invention; 
Fig. 6 schenrratically illustrates still another stalled virtual clock calendar queue implementation of this inverition; 
Fig. 7 schematically illustrates an ABR connection having a source-to-destination control loop; and 
Fig. 8 schematically illustrates an ABR connection having a segmented control loop. 



50 



55 



A. A Representative Environment 

Turning now to the drawings, and at this point especially to Fig. 1, the input and output ports of an ATM switch 21 
typically are coupled to one or more physcal layers via respective Utopia 2 interfaces and to a switch control processor 
module 22 via a second suitable interface. This enables the switch 21 to exchange data and control cells with any 
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physical layers that are connected to them, and to also exchange control cells with the control processor module 22. 
In keeping wrth standard practices, the communicalion channels are unidirectional, so a pair of channels are required 
for bl-directioral communications. 

The switch 21 comprises a switching fabric 24. a fabric control module 25 and a reservation ring 26 for switching 

5 data and control cells from input queues to per-VC output queues. The cells in these queues are stored in the data 
path in data memory 27. and these input and output queues are managed by a queue control rrKJdule 28. Typically, 
the data memory 27 is sized to hold up to roughly 12000 cells. Connection records for the data and control ceil flows 
are stored in the control path in control memory 29, together with certain types of control cells which are intercepted 
by a rate based engine and traffic multiplexer 31 for routing to the control processor module 22. Suitably the control 

10 RAM 29 is capable of accommodating up to about 8200 connection records and 64k cell records. The interaction of 
the control processor module 22 with the switch 21 is beyond the scope of the present invention and. therefore, is not 
described herein. However, persons who are familiar with ATM switch design will understand that the control processor 
is primarily responsible for pertorming connection establishment and termination, as well as 0AM (Operation and 
Maintenance) functions. 

'5 The data path of the switch 21 is synchronously clocked (by means not shown) at a predetermined rate of. say 

40 MHz. However, in keeping with conventional synchronous pipeline design practk;es. the phase of this clock signal 
is delayed (by means also not shown) by differing amounts at different points atong the data path to give the data 
adequate time to settle prior to being transferred from one pipelined stage to the next. 

In accordance with standard practices, a source wishing to communk:ate with a destinatran initiates negotiations 

20 with the ATM network within which the switch 21 reskJes by sending a SETUP message to the network. This message 
identifies the destination and explkiitly or impliedly specifies all of the relevant traffk; parameters for the requested 
connection. If the network is prepared to commit to the traffic contract which is defined by these traffic parameters {tr 
a modified versk^n of the parameters that the source is willing to accept), the network routes the SETUP message to 
the destination. Then, if the destination is ready to receive message traffic from the source in accordance with the 

25 terms of the traffic contract, the destinatkxi returns a CONNECT message to the source. This CONNECT- message 
confirms that a connectran has been established on a specified virtual circuit (VC) within a specified vii^ual path (VP) 
for a cell flow that conforms to the traffic contract. See ITU-T Recommendaton 0.2391 and ATM Forum UNI 4.0 Spec- 
ificatk>n. "Permanent" virtual connections can be established by provisioning, without invoking these signaling proto- 
cols. 

30 Data cells begin to flow after a connection is established. As shown in Fig. 2. the form of the cells change as they 

pass through the switch 21 because of the operations, the switch performs. Cells may be replicated within the switch 
21 for muttk:asting, but the following discussran will be limited to unicast operations to avoid unnecessary complexity. 

As indicated in Fig. 2 at 41 . each inbound cell that the switch 21 receives has a header containing a VP index and 
a VC index. These indices combine to define a unique address for one hop of the connection. A connection may be 

3S composed of multiple hops, so the VP and VC indces for the next hoop are written into the header of the cell as it 
passes through the switch 21 as indicated in Fig. 2 at 42. 

The switch 21 empksys the VP and VC indrces of the inbound cell (Fig. 2 at 41 ) to compute the address at which 
the connectbn record for the associated flow resides within the control RAM 29. Typically, this connection record 
includes a bit vector for identifying the output port (i.e., the switch-level "destination*) at which the flow exits the switch 

■^0 21 . a priority index for identifying the relative prwrity of the flow on a granular priority scale, and a circuit index (^Circuit 
Index") whch uniquely identifies the flow internally of the switch 21. As shown in Fig. 2 at 43, these connectk)n param- 
eters are written into the cell header Then, the ceil is written into the data RAM mennory 29, while a pointer to the cell 
is linked into an appropriate one of a plurality of FIFO input queues, where the selectkxi of the queue is based on the 
priority of the related flow 

^5 The relative prkjrities of the head of queue cells within these input queues are examined during each cell time, 

and the head of queue cell having the highest priority is selected for arbitration during the next arbitraton session. 
Furthermore, the priority <rf any kDwer priority head of queue cell (i.e.. any non-selected head of queue cell) is incre- 
mentally increased (by means not shown), thereby increasing the probability of that cell being selected for arbitration 
during the next arbitratk)n session. Therefore, even though the higher prkarity input queues have greater throughput 

50 per unit time than the lower priority queues, the kjwer priority queues have bounded delay because the priority of their 
head of queue cells increases as a function of time. 

Each arbitratkjn cycle requires one cell time of the switch 21 . so the routing informatksn for the cells that are selected 
for arbitration is fed into the reservation ring 26 one cell time prior to the release into the switching fabric 24 of the pay 
loads of the cell or ceils that win the art^itration. In other words, as shown in Fig. 2 at 44. the cells that are received by 

55 the resen/ation ring 31 and switching fabric 32 are composed of the headers of the cells for the next arbitratksn cycle 
(i.e.. the "current cells') foltowed by the bodies or paytoads of the celts successfully arbitrated during the previous 
arbitration cycle (i.e. . the 'prevbus cells'). Thus, when the cell bodies reach the fabric 32, the fabric is already configured 
by the fabric control 33 to route those cells to their respective output port destinatwn. For additional infomnation on the 
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reservation ring 31 and the artDitration it performs, see Cisneros. A.. 'Large Packet Switch and Contention Resolution 
'^^vtce.' Proceedings of the XIII Int ernational Switching Symposium, 1990, paper 14. Vol III pp 77-83 and Lyies U 
S.P. 5.519.698. which Issued May 21, 1996. 

In the illustrated embodiment, cells are decomposed into four bit wide nibbles for arbitration and routing. Thereafter 
(with the exception of 'idle cells" that may be provided for testing the switching processes) the cells are reassembled 
and queued in the data path, the control path, or both (a) for time scheduled transfer to the appropriate output ports 
of the switch 21 and/or (b) for transfer to the control processor module 22. The time scheduled transfer of cells to the 
output ports of the switch 21 is central to this invention so that subject is discussed in further detail hereinbelow On 
the other hand, the decomposition and reassembly of the cells, the testing processes, and the interaction of the control 
processor 22 with RM (Resource Management) and OAM (Operation and Maintenance) cells are incidental topics 
which need not be considered in depth. 

Referring to Fig. 3. it will be understood that the switch 21 fans out on the output or transmit side of the switching 
fabric 24. Thus, while one output channel of the switch 21 is shown, it will be understood that this channel is generally 
representative of the other channels. 

As shown, there suitably is a fill cells module 51 for accepting cell bodies and their associated circuit indices from 
the switching fabnc 24. The "effective cell time" on the output side of the switching fabric 24 is determined by the ratio 
of the nominal cell time to the "k" speed-up factor Thus, for example, if the nominal cell time is 11 3 clock cycles/cell, 
the effective cell time on the output side of the switching fabric 25 is 56.5 cycles/cell if k=2. 

When a valid cell is received, the fill cell module 51 typkally uses cell structures from a linked and numbered free 
list 52 of such data structures for writing the cell into the data memory 27. To that end, the fill cell module 51 suitably 
includes a fetch state machine 53 for fetching cell structures form the top of the free list 52 on demand. This enables 
the fill cell module 51 to insert the circuit index for the cell and a pointer to the locaton of the cell in data memory 27 
into an "arrivar message that it sends to notify a cell flow control unit 55 of the ceirs an-ival. The circuit index enables 
the flow control unit 56 to ascertain the VC or flow to which the cell betongs from the connection record in the control 
menrory 29. This, in turn, enables the cell flow control unit 55 to check the traffic shaping status of the flow. An OAM/ 
RM recognizer 57 advantageously is provided to enable the flow control unit 55 to identify these control cells and to 
determine whether they are to be queued in the data path, the control path, or both. 

The memory pointers for ceils of traffic contract compliant flows are queued in per-VC queues in response to 
"addCeir messages which the cell flow control unit 55 sends to a queue control unit 58. Each addCell message identifies 
the cell to whk^h it pertains and the circuit index for the associated flow or VC. Moreover, the addCell message also 
indicates whether the cell is to be queued in the data path, the control path, or both. When the cell has been appropriately 
queue, the queue control unit 58 returns an "added" message to the cell flow control unit 55 for notifying the flow control 
unit 55 that the newly queued cell needs to be taken into account during future rate shaping computations on the VC 
to which it belongs. 

Advantageously the queue control unit 58 monitors the length of the per-VC queues with respect to the depth 
control limits that are set on the respective queues. This enables the control unit 58 to initiate congestion control action 
on ABR flows when their per-VC queues become excessively tong. It also permits the control unit 58 to identify flows 
that are exceeding their traffic contract so that an appropriate polkiing function (not shown) can be invoked to drop or 
log cells of such non-compliant flows. 

An admission controller 61 monitors the "added* messages that are returned by the queue control unit 58 to^ause 
a scheduler 62 to schedule the head of queue cells for the non-empty per-VC queues on a calendar queue 63 for 
transmissk)n at scheduled times. Suitably the scheduler 62 employs a per-VC virtual clock to schedu^e these head of 
queue cells on the calendar queue 63 in accordance with respective virtual finishing times, VT(f, that the scheduler 
62 computes for them (or. alternatively, Virtual start times"). Please see Section lll.C.1 above. 

The calendar queue 63 tracks system 'real time* or "current time" to prevent any of the scheduled cells from being 
released for transmission prior to its scheduled time. In other words, the scheduler 62 and the calendar queue 63 
implement a stalled virtual clock so that the cells that are scheduled for transmission are released for transmission 
only when system real time has at least reached their respective scheduled transmissksn times. As illustrated, connec- 
tions having cells that have been released for transmisswn by the calendar queue 63 are linked into a link list of 
so connections that have cells ready for transmission on a transmit list 65. 

The calendar queue 63 notifies the flow control unit 55 whenever it releases a cell for transmission on any given 
connection. The flow control unit 55, in turn, requests the reference to the next cell (i.e., the new head of queue cell), 
if any, on the per-VC queue for the given connection and notifies the admisskjn controller 61 that it should admit this 
reference to the scheduler 62 for scheduling. Thus, the admission controller 61 effectively engages in closed loop 
communicatksns with the calendar queue 63 to ensure that the head of queue cells which it admits for scheduling 
thereon are admitted to the exclusion of all other cells in the per-VC queues. Thus, the calendar queue 63 may be 
implemented by employing one or more time bounded time-wheels or time lines,' 66. Please see Section 1 II C.4 above. 
The time span of these time-wheels must be at least as tong as the period of the towest frequency flows that the system 
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is designed to support to prevent time wrap induced ambiguities and preferabty is twice as Jong so relative times can 
be compared using two's complement computations. 

B. Shaping Flows of Rxed Bit Length Data Transport Limrts to Specffied Peak Flow Rates 

Referring to Fig. 4. it will be evident that stalled virtual clock transmission control is well suited for shaping time 
multiplexed flows of fixed bit length data transport units, such as ATM cells, from an output queued routing mechanism 
to specified peak data unit flow rates, such as PCRs for DBR/CBR ATM service. As previously described, the data 
transport units of the flows that are routed to a given output port are queued, after being routed, in per-flow queues. 
The data transport units at the heads of these queues then are admitted by an admissbn controller 61 (to the exclusion 
of all other transport units) for scheduling on a time line calendar queue 63 by a scheduler 62. The scheduler 62, in 
turn, performs per-flow virtual clock computations on these head of queue transport units to schedule them for release 
from the calendar queue 63 in accordance with their respective theoretical finish times. VT(f, j+1 ), or their respective 
theoretical start times. Please see Section liLC.4 above. 

Real time advantageously is incremented on the time line 63 at a rate that enables the shaped, time multiplexed 
output traffic to essentially fill the bandwidth ol the output link 71 . As will be recalled, the maximum number o1 resolvable 
time slots into which the scheduler 62 can map the head of queue members of the respective flows is based on the 
ratio of the maximum permissible frequency to the minimum permissible frequency of those flows. Thus, the rate at 
which real time is incremented from bucket-to-bucket at a ratranale multiple of the cell rate. 

Data transport units reskiing in time slots which represent times that are earner than or equal to the current real 
time of reference for the time line 63 are eligible for transmission and, therefore, are linked into a transport list*65 as 
previously described. However, those data transport units which reskle in time slots that are associated with later time 
slots of the time line 63 remain in a pending state until system real lime advances sufficiently to reach those time stots. 
To avoki rollover ambiguities, the time line 63 is designed to ensure that all references to earlier scheduled data transport 
limits are removed from each time slot before any references to later scheduled transport units are inserted therein in 
anticipation of the next scan. 

While the above-described arrangement effectively shapes conforming DBR/CBR ATM flows to the PCRs specified 
by their traffic contracts, it does not a\6 in bringing the Cell Delay N^riation (CDVs) of those flows into confomiity with 
the TpcR parameters of their traffic contracts. 

C. Multiple Priority Levels for Minimizing RelatWe CDV 



In accordance wit the present inventk)n. data transport units that are delivered to a multiplexing point, such as an 
output port of an ATM switch, by flows having different frequencies are prbritized so that the data transport units of 

35 the higher frequency flows are given transmit priority over any data transport units of lower frequency flows with which 
they happen to collide. As shown in Fig. 3. this transmit priority can be implemented by steering the data transport 
units that are admitted for scheduling by a stalled virtual ckxk scheduling mechanism 63 or the like to one or another 
of a plurality of priority rank ordered time-lines 66a-66e or output FIFO queues based on the frequencies of the flows 
to which those respective data transport units belong. For example, for an ATM switch, it may be advisable to implement 

40 on the order of five different frequency dependent/class of sen/ice dependent output priorities, including (1 ) a top.priority 
fa cells from flows that have negotiated output rates of at least 1/16 of the full rate of the output link (i.e.. its aggregate 
bandwkSth). (2) a second priority for cells form flows having negotiated output rates ranging from 1/16 to 1/256 of the 
output link rate, and (3) a third priority for cells from flows having negotiated output rates ranging from 1/256 to 1/4096 
of the link rate. The lower two prwrities then suitably are established for ABR connections that have non-zero negotiated 

^ MCR rates and for UBR connections and ABR connections that have MCR rates of 0, respectively. 

As Will be appreciated the present invention effectively reduces the CDVs of the higher frequency flows, without 
materially increasing the CDVs of the tower frequency flows. As a general rule, the CDV that is tolerable is a function 
of the negotiated rate for a flow. For example, a CDV of 1 00 cell limes is very large with respect to an expected emissran 
interval of one cell every 10 cells, but generally insignificant if the negotiated emission inten^al is only one cell every 

so 2029 cells. 

When a calendar queue mechanism is emptoyed to schedule the data transport units or cells of the different fre- 
quency flows for transmission, the high frequency high prrarity flows need to be resolved to the precision of a single 
cell time while being scheduled to achieve an acceptably low CDV. but the low frequency/low priority ftows can be more 
coarsely resolved to a precision of, say. 16 cell times. This means that the number of time slots on the calendar queue 
55 63 can be reduced. This enables the amount of memory that is required to implement the calendar queue 63 to be 
reduced at the cost of losing some typically unneeded precision in the scheduling of the head of queue cells of the 
lower frequency flows. 

It is to be understood that the frequency based prioritization technique which the present inventbn provkie for 
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resolving transmission conflicts at multiplex points among flows of different nominally Hxed frequencies may be em- 
ployed in many different applications for reducing the relative jitter of ttie flows, including in applications having work 
conserving per-flow output queues for feeding cells or other data transport units into such a multiplex point. 

Aoo!!"*.®".?'"*' *° PCR Paramelers for ABR Service In Networks Having Source-to-Destlnation 

ABR Control Loops 

As will be recalled, the standard ABR traffic contract contemplates explicitly or implicitly specified PGR and MCR 
parameters, where MCR may equal 0. An ABR flow that confomns to its GCRA (1/MCR. t^cr) is valid and entitled to 
service, but the network may not be giving the flow the bandwidth to which it is entitled. Contrastingfy. a flow which 
violates Its GCRA (1/PCR, tpcR is non-conforming. Thus. ABR connections or flows that are routed to the output of a 
network or network element, such as by the ATM switch 21, advantageous^ are shaped to ensure (1) that the ABR 
flows with non-zero MCR guarantees receive adequate output bandwidth to effectively satisfy those guarantees and 
(2) that none of the ABR flows violate their PCR commitments at such output. In networks that have multiple multiplex 
^5 points, rt IS useful to perform this shaping at each of the multiplex points. 

Fig. 4 illustrates one technk^ue for shaping ABR ftows or VC connecttons to conform them to their traffic contracts 
at an output ot a network or network element. As shown, the rates at which the respective flows are requesting output 
service are monitored: first to detemiine at 85 whether the ftows are or are not GCRA (1/MCR. t^^p) compliant and 
then to determine at 86 whether the GCRA (1/MCR, t^cr) non^ompliant flows are or are not GCRA (1/PCR Wp) 
compliant. Control signals reflecting the results of these cascaded tests 85 and 86 are fed back to the scheduler 62 
and rts associated steering logic 87 to permit appropriate rate shaping adjustments to be made to the rates at whk:h 
the data transport units (e.g.. cells) of the respective flows are fed to the output. As will be appreciated the desired 
state IS for all of the ABR ftows tobe GCRA (1/MCR, x^^) non^ompliant (or just barely compliant) and GCRA (1/PCR 
Tpcn) false/compliant because those algorithms test for minimum and maximum acceptable flow rates respectively ' 
More particularly. GCRA (1/MCR. x^ch) compliant flows may not be receiving the output bandwidth that they need 
to satisfy their MCR guarantees. Therefore, when such a flow is identified at 85, the scheduler 62 and its steering togk: 
87 are set-up to schedule subsequently received references to data transport units (e.g. . cells) of that ftow on a relatively 
htgh priority stalled virtual clock calendar queue 88 for output at a rate somewhat in excess of its guaranteed MCR 
For example, these subsequently received cell or packet references suitably are scheduled on the calendar queue 88 
for emission at a rate that is determined by multiplying the MCR for the flow by a predetermined speednjp factor This 
causes the network to give these additional potentially 'lagging- flows the additional output bandwidth that is needed 
to satisfy their MCR guarantees. 

GCRA (1/MCR. t„cR) non-compliant ABR flows are further tested at 86 to determine whether they are demanding 
output servrce at acceptably low or unacceptably high GCRA (1/PCR. tpcp) compliant or non-compliant rates, respec- 
tively. The results of this test are captured by another control signal whch is fed back to the scheduler 62 and its 
steenng togic 87 to set them up for rate sensitive enqueuing of the subsequently received references to these GCRA 
(1/MCR. Imcr) non-compliant flows on a lower prtority queue 89. Specifically the subsequently received references 
to a flow that is found to be GCRA (1/PCR, Tpcp) complant at 86 are appended to the tail of the transmit list for the 
ABR flows with MCR guarantees for it is servced in round-robin order. Contrastingly, when a flow is found to be GCRA 
(1/PCR. Tpcr) non-compliant at 86, the subsequently received references for that flow are scheduled by the scheduler 
62 on a non-work conserving, stalled virtual clock calendar queue 89 in accordance with the PCR for that flow, thereby 
imposing a PCR limit on the flow. 

More generally, it will be evdent that the embodiment of Fig. 4 implements the flowing ABR traffic shaping algorithm: 

If GCRA (1/MCR. Tmcr) compliant then enqueue in high priority queue at rate MCFTspeedup 
else if GCRA (1/PCR, tpcR) compliant then enqueue in low-priority wori< conserving queue 
else enqueue in tow-priority non-work conserving queue based on PCR emission interval. 

However, alternatives that are based on similar and different principles will suggest themselves. 
For example, as shown in Fig. 5. one potentially attractive altematVe is to enqueue the references to the data 
transport units or cells of flows that are found to be GCRA (1/MCR. Tmcr) non-compliant and GCRA (1/PCR. ipcR) 
compliant at 85 and 86 (Fig. 4). respectively, in both (1) the high priority calendar queue 88 at a MCR scheduling 
interval, and (2) the work consenting region of the lower prtority queue. This creates a Vace" conditton because when- 
ever one instance of such a dual queued reference is reached for service in either of the queues, a delinker 91 rerrwves 
the other instance of that reference from the other queue. This means that a tighter bound may be maintained on the 
tolerance, t. The traffic shaping algorithm of this modified embodiment is: 
At enqueue: 
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enqueue in high priorrty queue wrth rate MCR; 

it GCRA (1/PCR. tpcn) compliant then enqueue in lew-priority work conserving queue; 
else enqueue in low-prbrity non-work conserving queue based on PGR emission inten/at: 
At dequeue: 

5 \i dequeuing form high priority queue then delink from tow priority queue else delink from high priority queue. 

Fig. 6 illustrates still another alternative for shaping ABR connectk>ns or flows. In this embodiment, the output 
bandwidth that is provided for each MCR non<ompliant ABR flow that has a non-zero MCR guarantee Is dynamically 
adjusted based on the average spacing between ABR cells on the output link (i.e., at the transmit side of a line interface) 

10 and the current length of the ABR "ready queue/ The current length. Len. of this ABR "ready queue" conveniently is 
determined by measuring the length of the work conserving queue on the ABR transmit lists 65e and 65f as at 93. On 
the other hand, the average spacing, S. between ABR cells on the calendar queue can be detennining at 94 the percent 
of the link by divkiing the bandwidth that is available for ABR flows after altowance is made for any non-ABR flows. 
The bandwidth required by these non-ABR flows suitably is determined by computing at 94 the aggregate bandwidth 

'5 needed to sen^ice the transmit list 65 entries for the non-ABR flows. 

For example, it it is determined at 94 that 50% of the bandwidth of the output link is available for the ABR flows, 
it follows that the average space, S. between ABR cells or data transport unrts on the output link then is two cell times. 
This inforrmtion. together with the computed length, Len, of the ABR 'ready queue.* enables the service for the MCR 
non-comptiant ABR flows to be optimized by having the scheduler 62 schedule the incoming references for those flows 

20 on the non-work conserving calendar queue 89 at respective emisskxis inten/als T, of: 



T, = Min (Sj, Max (P,. R)) (3) 

Where: .'.^ 

R = an estimated round robin ABR 

sen^ice time (R = current time + S*Len); 
P, = the earliest servrce contract compliant departure for the next cell of flow t (P, = current time + 1/PCR); and 
^0 St = the latest service contract compliant departure for the next cell of fk>w t {S^ = current time + l /MCR) 

E. Traffic Shaping lor Segmented ABR Connections 

As is known, ABR connections may operate with source-to-destination control loops, such as shown in Fig. 7 at 
101 . or with segmented control loops, such as shown in Fig. 8 at 102. When segmented control loops are emptoyed, 
each of the separately controlled segments functions as a virtual source, VS. for the next segment. Thus, each ABR 
control segment (except the first) is sourced by a virtual source which assumes the behavior of an ABR source endpoint. 
Backward resource management (RM) cells are removed from the control loop when they are received by a virtual 
source. tHowever. each source or virtual source can compute an estimate of the available network bandwidth to its 
destination or virtual destination, VD. based on this feedback which, in turn, can be expressed as an Albwed Ceil Rate 
(ACR) on a per-segment basis. Therefore, ABR traffic shaping advantageously is carried out segment-by-segment for 
these ABR connections with segmented control loops, using a maximum aitowable per-flow data transport unit or cell 
rate of Max(MCR, Min(ACR, PCR). The embodiments of Figs. 4-6 can implement such traffic shaping by setting PCR' 
equal to Max(MCR, Min(ACR, PCR) and then by using PCR in place of PCR in the computations. 

45 



Clatme 

1 . A process for shaping time multiplexed serial flows of packets to respective available bit rate network traffic con- 
tracts in a network having a segmented control kx>p (102) which is configured to provkle segment-by-segment 
congestion control for said network; said traffic contracts establishing respective minimum and peak packet emis- 
sion rates for said flows and respective tolerances on said rates; said process comprising 

analyzing congestion control activity within the respective segments of saki control loop (102) for estimating 
how much bandwidth the respective segments of said network are able to provide for available bandwidth 
traffic: 

emptoying said bandwidth estimates to compute allowed packet rates for the respective available bit rate flows 
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in the respective segments of said network 

refining the peak packet emissbn rates for the respective flows in the respective segcTients of sakd network 
by setting said rates to be approximateV equal to respective limiting values without materially altering the 
tolerances on said rates: where the limiting value for the refined peak packet rate for any given one of sakJ 
flows in any given one of said segments has an upper bound approximately equal to the lesser of the contrac- 
tually established peak packet rate for the given flow and the allowed packet rate for the given flow in the given 
segment of saKj network and a kswer bound approximately equal to the contractually established minimum 
packet rate for the given flow; 

controlling the rates at which packets of the respective flows are emitted into the respective segments of sakd 
networks by selectively enqueuing the packets of any gwen flow for emission into any given segment of said 
network (i) on a non-work consen/ing calendar queue when the rate at which packets of the given flow are 
flowing in the given segment of the network is non-compliant with the traffic contract for the given flow as 
modified by the refined peak packet rate and (ii) on a work conserving queue when said rate is compliant with 
said contract. 

2. A process as claimed in claim 1 . wherein 

packets of said given flow are scheduled on a first non-work consenting calendar queue for emission into said 
given segment of said network at a rate in excess ot the contractual^ established minimum emission rate for 
sa^ flow when the rate at which sakJ packets are flowing in said given network segment is out of tolerance 
with respect to said minimum rate; 

packets of said given flow are scheduled on a second non-work consenting calendar queue for emission int<5 
said network segment at a rate approximately equal to the refined peak emission rate for said f!ow in said 
segment when the rate at which said packets are flowing in sakJ segment is out of tolerance with respect to 
said refined peak rate; 

said first non-work consen/ing calendar queue (88) has higher priority than said work consenting queue for 
resolving scheduling conflk;ts therebetween, and said second non-work consenting calendar queue is of in- 
deterministic priority relative to said first queue and said work consen/ing queue. 

30 3. A process as claimed in claim 2. wherein 

packets of said given flow also are enqueued on said work consen/ing queue when the rate at which said 
packets are flowing in said given segment is out of tolerance with respect to said minimum rate; 
packets enqueued on both said first calendar queue and sakJ work consenting queue are delinked from one 
of those queues upon being dequeued from the other, thereby resolving the race conditions that are established 
by such enqueuing. 

4. A process as claimed in claims 2 or 3, wherein 
said first and said second calendar queues are respective linked, real time indexed, data structures. 

5. A process as claimed in claim 4. wherein 
saki second calendar queue and said work consenting queue are dynamically reallocated segments of one 

of said data stojctures, where the reallocation of such data stnjcture segments is controlled by a pointer which 
cyclically traverses said data stmcture as a function of real time. 

6. A process as claimed in claim 5, wherein 
said pa:kets are of uniform, fixed bit length. 

7. A process as claimed in claim 6. wherein 
packets are dequeued from said work consenting queue in accordance with a strategy which provides ap- 
proximate round robin sentrce for packets enqueued on said work conserving queue. 

8. A process as claimed in claim 7, wherein 
said packets are fixed byte length cells for asynchronous transfer mode communicatwns, 

9. A process as claimed in any one of claims 1 to 8. the process being used in a packet switched communication 
system. 
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